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A search for the neutrinoless, lepton-flavor violating decay of the tau lepton into three charged 

leptons has been performed using 376 fb~^ of data collected at an e^e~ ccntcr-of-mass energy 
around 10.58 GeV with the BABAR detector at the PEP-II storage rings. In all six decay modes 
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considered, the numbers of events found in data are compatible with the background expectations. 
Upper Umits on the branching fractions are set in the range (4 — 8) x 10~* at 90% confidence leveL 

PACS numbers: 13.35.Dx, 14.60.Fg, ll.30.Hv 



Lepton-flavor violation (LFV) involving charged lep- 
tons has never been observed, and stringent experimental 
limits exist from muon branching fractions: B{ii — *■ ej) < 
1.2 X 10-" [H and B{n eee) < 1.0 x IQ-^^ [| at 90% 
confidence level (CL). Recent results from neutrino oscil- 
lation experiments show that LFV does indeed occur, 
although the branching fractions expected in charged lep- 
ton decays due to neutrino mixing alone are probably no 
more than 10"" 

In tau decays, the most stringent limit on LFV is 
B{t fi-f) < 4.5 x 10-8 at 90% CL @. Many de- 
scriptions of physics beyond the Standard Model (SM), 
particularly models seeking to describe neutrino mixing, 
predict enhanced LFV in tau decays over muon decays 
with branching fractions from 10"^° up to the current 
experimental limits 0, 0, Q • An observation of LFV in 
tau decays would be a clear signature of non-SM physics, 
while improved limits will provide further constraints on 
theoretical models. 

This paper presents a search for LFV in the neutrino- 
less decay t~ — > £~£~^£~ , where £ is an electron or muon. 
All possible lepton combinations consistent with charge 
conservation arc considered, leading to six distinct de- 
cay modes (e-e"'"e-, ^^^"e" , fi~e^e~ , e'^^^fj,^ , e^/i+zi", 
[9|]. The analysis is based on data recorded 
by the BaBAR detector at the PEP-II asymmetric-energy 
e~^e~ storage rings operated at the Stanford Linear Ac- 
celerator Center. The data sample consists of 339 fb^^ 
recorded at ^/s — 10.58 GeV, and 37fb~^ recorded at 
yfs = 10.54 GeV. With an expected cross section for tau 
pairs at t he luminosity-weighted y^s of CTtt — 0.919 ± 
0.003 nb [10], this data sample contains about 690 mil- 
lion tau decays. 

The BaBAR detector is described in detail in Ref. 
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Charged-particle (track) momenta are measured with 
a 5-layer double-sided silicon vertex tracker and a 40- 
layer helium- isobutane drift chamber inside a 1.5-T su- 
perconducting solenoidal magnet. The transverse mo- 
mentum resolution is parameterized as aprp/pT — (0.13 • 
PT/[GeV/c] + 0.45)%. An electromagnetic calorimeter 
consisting of 6580 CsI(Tl) crystals is used to identify 
electrons and photons, a ring- imaging Cherenkov detec- 
tor is used to identify charged hadrons, and the instru- 
mented magnetic flux return (IFR) , embedded with lim- 
ited streamer tubes and resistive plate chambers, is used 
to identify muons. 

A Monte Carlo (MC) simulation of lepton-flavor violat- 
ing tau decays is used to optimize the parameter space for 
the search. Simulated tau-pair events including higher- 
order radiative corrections are generated using KK2f [3| 
with one tau decaying to three leptons with a 3-body 



phase space distribution, while the other tau decays ac- 
cording to measured rates [l3| simulated with Tauola 
[3]. Final state radiative effects are simulated for all 
decays using Photos I l5j. The detector response is simu- 
lated with GEANT4 [16| , and the simulated events are then 
reconstructed in the same manner as data. 

The signature of the decay t~ £~£'^£^ is a set of 
three charged particles, each identifled as either an elec- 
tron or muon, with an invariant mass and energy equal 
to that of the parent tau lepton. Candidate signal events 
in this analysis are required to have a "1-3 topology," 
where one tau decay yields three charged particles, while 
the second tau decay yields one charged particle. Events 
with four well-reconstructed tracks and zero net charge 
are selected, and the tracks are required to point toward 
a common region consistent with t~^t~ production and 
decay. The polar angle of all four tracks in the laboratory 
frame is required to be within the calorimeter acceptance 
range. Pairs of oppositely-charged tracks are ignored if 
their invariant mass, assuming electron mass hypotheses, 
is less than 30 MeV/c^ , as these tracks are likely to be 
from photon conversions in the traversed material. The 
event is divided into hemispheres in the e^e~ center- 
of-mass (cm.) frame using the plane perpendicular to 
the thrust axis, as calculated from the observed tracks 
and neutral energy deposits. The signal hemisphere must 
contain exactly three tracks while the other hemisphere 
must contain exactly one. 

Each of the charged particles found in the signal hemi- 
sphere must be identified as either an electron or muon 
candidate. Electrons are identified using the ratio of 
calorimeter energy to track momentum (E/p), the ioniza- 
tion loss in the tracking system (dE / da;) , and the shape of 
the shower in the calorimeter. Muon identification makes 
use of a neural net, inputs to which include the num- 
ber of hits in the IFR, the number of interaction lengths 
traversed, and the energy deposition in the calorimeter. 
Muons with momentum less than 500 MeV/ c do not pene- 
trate far enough into the IFR to be identified. For the lep- 
ton momentum spectrum predicted by the signal MC, the 
electron and muon identification requirements are found 
to have an average efficiency per lepton of 91% and 65%, 
respectively. The probability for a pion to be misidenti- 
fied as an electron in 3-prong tau decays is 2.7%, while 
the probability to be misidentified as a muon is 2.9%. 

The particle identification (PID) requirements are not 
sufficient to suppress certain backgrounds, particularly 
those from light quark pair production and higher-order 
radiative Bhabha and /i^/i^ events that can have four 
leptons in the final state. To reduce these backgrounds, 
additional selection criteria are applied to the six differ- 
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ent decay modes. For all decay modes, the momentum of 
the 1-prong track is required to be less than 4.8 GeV/c 
in the cm. frame. Additionally, the track in the 1-prong 
hemisphere is assigned the most-likely mass hypothesis, 
and the mass of the 1-prong hemisphere is calculated 
from the four-momentum of that track and the missing 
momentum in the event. This mass is required to be 
in the range 0.3 — 3.0 GeV/c^ for all channels except 
e~e^e^ and /i^e+e^, for which the mass is required to 
be in the range 0.5 — 2.5 GeV/c^. For the e~e~^e~ and 
/Lt~e"'"e~ decay modes, radiative Bhabha events are further 
suppressed by rejecting events with pairs of oppositely- 
charged electron tracks in the 3-prong hemisphere with 
invariant mass less than 250 MeV/c^. For the e~e+e^ 
and e^fi^fi^ decay modes, the charged particle in the 
1-prong hemisphere is required to deposit energy in the 
calorimeter, and must not be identified as an electron, 
while for the /i^e+e^ and decay modes this track 

must not be identified as a muon. For the e~e~^e^ and 
e~iX^li~ decay modes, the net transverse momentum of 
the four tracks must be greater than 400MeV/c, while 
for the /i~e+e~ mode it must be greater than 200MeV/c. 
Events in all six decay modes are required to have no 
track in the 3-prong hemisphere that is consistent with 
being a kaon. 

To reduce backgrounds further, candidate signal events 
are required to have an invariant mass and total energy 
in the 3-prong hemisphere consistent with a parent tau 
lepton. These quantities are calculated from the observed 
track momenta assuming lepton masses that correspond 
to the specific decay mode. The energy difference is de- 
fined as AE = E*^^ — i^bcamj where E*^^ is the total 
energy of the tracks observed in the 3-prong hemisphere 
and -Eboam beam energy, with both quantities mea- 

sured in the cm. frame. The mass difference is defined 
as AAf = A/roc — ?7ir where Mice is the reconstructed in- 
variant mass of the three tracks and to^ = 1.777 GeV/c^ 



is the tau mass 13 1 



The signal distributions in the (AM, /S.E) plane (see 
Fig.[T|) are broadened by detector resolution and radiative 
effects. In all decay modes, the radiation of photons from 
the incoming e+e^ particles and from the outgoing tau 
particles leads to a tail at low values of Ai?. Radiation 
from the final-state leptons, which is more likely for elec- 
trons than muons, produces a tail at low values of AAf 
as well. Rectangular signal regions are defined separately 
for each decay mode. The signal region boundaries are 
chosen to provide the smallest expected upper limits on 
the branching fractions in the background-only hypothe- 
sis. These expected upper limits are estimated using only 
MC simulations and data control samples, not candidate 
signal events. For all six decay modes, the upper right 
corner of the signal region in the (AM, /S.E) plane is fixed 
at (20,50), while the lower left corner is at (—50,-200) 
for the /Li~e+e~ and e~/x+^~ decay modes, (—70,-200) 
(-100,-350) for /i+e-e", (-50,-200) for 



e~^fi~fi~, and (-20,-200) for ^~^+^~. All values are 
given in units of (MeV/c^, MeV). Fig. [T] shows the ob- 
served data in the (AM, A£^) plane, along with the signal 
region boundaries and the expected signal distributions. 
To avoid bias, a blinded analysis procedure was followed 
with the number of data events in the signal region re- 
maining unknown until the selection criteria were final- 
ized and all cross checks were performed. 

There are three main classes of background remaining 
after the selection criteria are applied: low multiplicity qq 
events (mainly continuum light-quark production) ; QED 
events (Bhabha and /Lt^/i"); and SM t~^t~ events. These 
three background classes have distinctive distributions in 
the {AE, AM) plane. The qq events tend to populate the 
plane uniformly, while QED backgrounds are restricted 
to a narrow band at positive values of AE, and t+t~ 
backgrounds are restricted to negative values of both AE 
and AM . A negligible two-photon background remains. 

The expected background rates for each decay mode 
are determined by fitting a set of probability density func- 
tions (PDFs) to the observed data in the grand sideband 
(GS) region of the {AE,AM) plane. The GS region, 
shown in Fig. [U lies between —600 and 400 MeV/c^ in 
AM and -700 and 400 MeV in AE, excluding the signal 
region. For the qq background, a PDF is constructed 
from the product of two PDFs Pm' and Pe' , where 
Pm'(AM') is a bifurcated Gaussian and Pe'[AE') = 
{l-x/y/l + x'^){l + ax + bx'^ + cx^) with a: = {AE'-d)/e. 
The {AM',AE') axes have been slightly rotated from 
[AM, AE) to take into account the observed correlation 
between AE and AM for the distribution. The resulting 
PDF has a total of eight fit parameters, including the ro- 
tation angle, all of which are determined by fits to MC qq 
background samples for each decay mode. For the t"*"t~ 
background PDF, the function Pm"{AM") is the sum of 
two Gaussians with common mean, while the functional 
form of Pe"{AE") is the same as that for the qq PDF. 
To properly model the wedge-shaped distribution due to 
the kinematic limit in tau decays, a coordinate trans- 
formation of the form AM" = cos/3i AM -I- sin/3i Ai? and 
AE" — cos/32 Ai? — sin/32 AM is performed. In total there 
are 11 free parameters describing this PDF, and all are 
determined by fits to the MG t+t" samples. 

For the three decay channels in which there is a signif- 
icant QED background, an analytic PDF is constructed 
from the product of a Grystal Ball function 



in 



AE' and a third-order polynomial in AM', where again 
the {AM' , AE') axes have been rotated slightly from 
{AE, AM) to tit the observed distribution. The six pa- 
rameters of this PDF, including the rotation angle, are 
obtained by fitting data control samples that are en- 
hanced in Bhabha or fJ-^ fJ.~ events. 

With the shapes of the three background PDFs deter- 
mined, an unbinned maximum likelihood fit to the data 
in the GS region is used to find the expected background 
rate in the signal region, shown in Table [D The PDF 
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FIG. 1: Observed data shown as dots in the {AE, AM) plane 
and the boundaries of the signal region for each decay mode. 
The dark and light shading indicates contours containing 50% 
and 90% of the selected MC signal events, respectively. 



TABLE I: Efficiency estimates, number of expected back- 
ground events (A^bgd), expected branching fraction upper lim- 
its at 90% CL (ULgQ^), number of observed events (A^obs), and 
observed branching fraction upper limits at 90% CL (UL90'') 
for each decay mode. All upper limits are in units of 10~*. 



Mode 


Eff. [%] 


Abgd 




Aobs 


TTT 




8.9 ±0.2 


1.33 ±0.25 


4.9 


1 


4.3 


li~e^e~ 


8.3 ± 0.6 


0.89 ±0.27 


5.0 


2 


8.0 




12.4 ±0.8 


0.30 ±0.55 


2.7 


2 


5.8 




8.8 ±0.8 


0.54 ±0.21 


4.6 


1 


5.6 




6.2 ±0.5 


0.81 ±0.31 


6.6 





3.7 




5.5 ±0.7 


0.33 ±0.19 


6.7 





5.3 



shape determinations and background fits are performed 
separately for each of the six decay modes. 

The efficiency of the selection for signal events is es- 
timated with a MC simulation of lepton-flavor violating 
tau decays. About 40% of the MC signal events pass the 
1-3 topology requirement. The total efficiency for signal 
events to be found in the signal region is shown in Ta- 
bleUfor each decay mode and ranges from 5.5% to 12.4%. 
This efRciency includes the 85% branching fraction for 1- 
prong tau decays. 

The PID efficiencies and misidentification probabilities 



have been measured with control samples both for data 
and for MC events, as a function of particle momentum, 
polar angle, and azimuthal angle in the laboratory frame. 
The systematic uncertainties related to PID have been es- 
timated from the statistical uncertainties of the efficiency 
measurements and from their discrepancies between data 
and Monte Carlo, and range from 2.3% for e~e~^e~ to 
12.5% for [ll|. The modehng of the tracking 

efficiency contributes an additional 1% uncertainty. All 
other sources of uncertainty in the signal efficiency are 
found to be small, including the statistical limitation of 
the MC signal samples, modeling in the generator of ra- 
diative effects, track momentum resolution, trigger per- 
formance, observables used in the selection criteria, and 
knowledge of the tau 1-prong branching fractions. The 
signal efficiency has been estimated using a 3-body phase 
space model and no additional uncertainty is assigned 
for possible model dependence. Despite the effect of slow 
muons on the PID efficiency, the total selection efficiency 
is found to be uniform within 20% across 95% of the 
phase space for the three leptons. 

Since the background levels are extracted directly from 
the data, systematic uncertainties on the background es- 
timation are directly related to the background param- 
eterization and the fit technique used. The finite data 
available in the GS region to determine the background 
rates contributes a significant uncertainty in all decay 
channels. Uncertainties related to the background PDFs 
are estimated by varying the background shape parame- 
ters within their errors and repeating the fits, and from 
changing the functional form of the PDFs. The total 
uncertainties on the background estimates are shown in 
Table HI Cross checks of the background estimation are 
performed by considering the number of events expected 
and observed in sideband regions immediately neighbor- 
ing the signal region for each decay mode. 

The numbers of events observed (iVobs) and the back- 
ground expectations (iVbgd) are shown in TableU with no 
significant excess found in any decay mode. Upper lim- 
its on the branching fractions are calculated according to 
Bfo^ = N^l/{2eLarr), where N^l is the 90% CL upper 
limit for the number of signal events when iVobs events 
are observed with A^bgd background events expected. The 
values e, L, and CTt-t are the selection efficiency, lumi- 
nosity, and r+r^ cross section, respectively. The un- 
certainty on the product Larr is 1.0%. The branching 
fraction upper limits are calculated including all uncer- 
tainties using the technique of Cousins and Highland [l3| 
following the implementation of Barlow [i^] ■ The sensi- 
tivity or expected upper limit ULgg"^, defined as the mean 
upper limit expected in the background-only hypothesis, 
is included in Table HI The 90% CL upper limits on 
the T~ £^£^£^ branching fractions are in the range 
(4 — 8) X 10^^. These limits represent up to an order of 
magnitude improvement over the previous experimental 
bounds (ill,!!!. 
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